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Abstract 

We extend a recently introduced class of exactly solvable models for recurrent neural networks 
with competition between ID nearest neighbour and infinite range information processing. 
We increase the potential for further frustration and competition in these models, as well 
as their biological relevance, by adding next-nearest neighbour couplings, and we allow for 
modulation of the attractors so that we can interpolate continuously between situations with 
different numbers of stored patterns. Our models are solved by combining mean field and ran- 
dom field techniques. They exhibit increasingly complex phase diagrams with novel phases, 
separated by multiple first- and second order transitions (dynamical and thermodynamic 
ones), and, upon modulating the attractor strengths, non-trivial scenarios of phase diagram 
deformation. Our predictions are in excellent agreement with numerical simulations. 

PACS: 87.30, 05.20 
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1 Introduction 



In real (biological) recurrent neural networks, where information processing is based on the cre- 
ation and manipulation of attractors, one typically observes an intricate interplay and compe- 
tition between long-range information processing (via excitatory pyramidal neurons) and short- 
range information processing (via short-range pyramidal neurons and inhibitory inter- neurons). 
Studying those properties of such systems which are linked to their spatial structure, using sta- 
tistical mechanical techniques, requires moving away from the more traditional infinite range 
models of attractor neural networks ffl, Q. With the latter objective, an alternative type of 
attractor neural networks was recently proposed and studied ||, in which neurons (represented 
by Ising spins) are mutually connected by a combination of infinite range synaptic interactions, 
and one-dimensional nearest-neighbour interactions. Although real biological network archi- 
tectures are obviously far more complex, such models, which are still sufficiently simple to be 
solved exactly, via a combination of mean-field techniques (as in e.g. [§]) and random field 
techniques (as in e.g. [||, ||, ||), would appear to represent a small but welcome step towards 
biological reality. Moreover, from a statistical mechanical perspective, the solutions of these 
models exhibited a remarkably rich behaviour, even in the so-called low storage regime (where 
the number of patterns stored in the interactions remains finite in the thermodynamic limit), 
and particularly in those regions in parameter space where the two types of interactions (long 
range versus nearest-neighbour) compete most strongly. The phase diagrams where found to 
describe a series of regions with different numbers of ergodic components, separated by both 
second and first order transitions (representing various dynamical transitions, in addition to the 
thermodynamic ones), and to increase dramatically in complexity with the number p of stored 
patterns. 

The present paper is devoted to a further exploration and enlargement of the class of models 
introduced in Q. We study two orthogonal extensions, each with their own specific objectives. 
Our first extension is to include also neuronal interactions between next-nearest neighbours in 
the ID-chain (in addition to the mean- field and nearest neighbour ones), and to study their 
impact on the phase diagrams. Here the motivation is, again, partly biological: short-range 
pyramidal neurons are believed to act on shorter distances than short-range inter-neurons, and 
simple models of the type proposed here have indeed been used recently to explain properties 
of the mammalian visual system [0, |j. We find, especially when the parameters controlling 
the new interactions are chosen such as to introduce further competition and frustration into 
the network, new phases are being created and the complexity of the phase diagram is again 
significantly increased. Our second extension is primarily motivated by our desire to understand 
the significant qualitative modification of the phase diagrams as observed in || resulting from 
just a small increase in the number of stored patterns (e.g. p = 1 versus p = 2). More specifically, 
in contrast to the traditional long-range models, in models with short-range interactions one finds 
a stronger disruptive effect of non-condensed patterns on the recall of the condensed ones. In 
order to shed light on such phenomena we extend the original models of || by modulating the 
embedding strengths of the individual stored patterns (and therefore the attractors themselves), 
as in ||, so that we can smoothly interpolate between, for instance, the p = 1 and p = 2 models. 
This reveals, as was expected on the basis of the qualitative differences between the p = 1 and 
p = 2 diagrams, a very complicated and interesting scenario of phase diagram deformation. 
Both extensions of the original models in Q introduce technical complications, but these are 
largely of a quantitative nature, and the extra work needed to again arrive at exact solutions is 
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more than adequately compensated by the richness of the resultant phase diagrams. 



2 Definitions 

As in ||, each of our extended models is defined as a collection of N binary neuron variables 
(i.e. Ising spins) a = (0*1, . . . , <tjv), with cij 6 { — 1, 1}, which evolve in time stochastically and 
sequentially, following the Glauber-type rule 

' 1 ±tanh[/?/i;(er(t))] 



Prob [<n(t + 1) = ±1] 



hi{rr) = ^2Jij(Jj (1) 



The parameters J™ represent the synaptic interactions, and the parameter f3 = 1/T controls 
the amount of stochasticity in the dynamics. If the interaction matrix is symmetric, the process 
(0) leads to a unique equilibrium state of the Boltzmann type, i.e. with microscopic state 
probabilities of the form Poo(°") ~ exp[— (3H(<r)] and with the conventional Ising Hamiltonian 
H(a) = — J2i<j cr iJij 'j- Information processing in such systems is based on the creation and 
manipulation of attractors in the system's configuration space, by a suitable choice of the spin- 
interactions {Jij}, which shape the energy landscape. In statistical mechanical terms, the two 
key aspects of these interactions which determine the analytical solvability or otherwise of the 
the resulting models are (i) the spatial structure defined by the interactions (reflected in which 
of the Jij are non-zero) , and (ii) the actual values taken by the non-zero interactions (which will 
generally be non-trivial, in order to achieve the objective of the creation of specific attractors). 
For the interaction matrix Jij we now make two different choices which both generalise the 
model class of Q, but in qualitatively different ways. 

Our first generalisation focuses on the values of those interactions which are present, while 
retaining the mean-field plus nearest neighbour interaction structure of 0: 



v 

model I : Jij = 



J: 



N 



(2) 



in which the components 6 { — 1,1} are all drawn independently at random, with equal proba- 
bilities. Neural networks of this type correspond to the result of having stored in a Hebbian-type 
fashion a set of p binary patterns £i = (£},■■■ , £f ) £ {—1, 1}^. The neurons can be thought of as 
arranged on a one dimensional array with mean-field interactions between all pairs given 
by N^ 1 J2i_ t JfiCiCji m combination with interactions between nearest neighbours neighbours 

+ 1) of strength J^i^i+i- We will only consider the case where limTv^oo p/N = 0. The 
parameters J* and control the embedding strength of pattern [i in the short- and long-range 
synapses, with negative values corresponding to the creation of 'repellors' rather than attractors. 
For uniform embedding strengths, J* = J s and J^ = Jg for all fjL, we recover ||. 

Our second generalisation affects the spatial structure of the system, rather than the prop- 
erties of the attractors (although the latter will be indirectly affected). Here our choice of 
interactions is 



model II : Jij 



E (3) 

u=l 



In neural networks of type II, the short-range synaptic interactions reach beyond nearest- 
neighbours; here Jf,Js ,Js € 3ft control the strengths of long-range, nearest-neighbour and 
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second nearest-neighbour interactions. Alternatively, in these models the neurons can be thought 
of as lying on a strip, mutually coupled by infinite range interactions of strength J^/N^ ■ £ in 

combination with short-range 'diagonal' interactions of strength Js arm 'edge' interac- 

(2) 

tions of strength J s • Ci+i ( see the figure). Note that the models of type II reduce to those 
in |J for Jg 2 ^ = 0. The most relevant observables in our models are the so-called overlap order 




parameters, defined as m^(<j) = N^ 1 J2i£,i a i> which measure the degree of similarity between 
the actual network state <x and the /i-th stored pattern. 

Due to the presence of short-range interactions in the above models (and, similarly, those 
of Q), the solution of even the simplest scenario where p -C N is already significantly more 
complicated than solving the standard infinite-range (Hopfield-type, j|, g]) cases. The solution 
of both models will be based on a suitable adaptation of the random- field techniques of § . 

3 Physics of Model I 

3.1 Solution via Random Field Techniques 

In order to find the phase diagrams we first isolate the p overlap order parameters by inserting 
1 = / dm 5[m — with m = (mi, . . . , m p ) and £j = . . . , £f ) in the expression for 

the asymptotic free energy per site / = — lirn/v->oo(/?A0 _1 In Z, and subsequently replace the 
delta functions by their integral representations. For model I this leads to 

/ = _ run _Lln / dmdrh e -/^(m,m) 

0jv(m, m) = -im ■ m - - ^ jjmj^ - — In R N (m) (4) 

where the non-trivial part of the calculation, mainly induced by the short-range interactions, 
has been concentrated in the term R^(rh) (we consider non-periodic boundary conditions): 

R N (rh) =J2U T ^+i e-^™^", T aiai+1 = e ^T.^^+PY.^'MO^^) 
cr i=i 

(5) 
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In the limit N —* oo the above integral will be evaluated via steepest descent. This results in 
an expression for / in terms of the relevant saddle-point of the asymptotic form of <pN'- f = 
extr m ^ lirrijv^oo <t>N ( m , r "-)- Since the quantity R(fh) does not contain the order parameters 
m, we can immediately take derivatives in (Q) with respect to m, which allows us to eliminate 
the conjugate variables rh via —irh^ = J^m^ for all /i. Furthermore we observe that, since 
for each /i the order parameter is coupled to the infinite-range embedding strengths J~, 
the so-called 'pure state' ansatz m = (m, 0, . . . , 0) will automatically render the solution of the 
model independent of for all \x > 1. From now on we will therefore use the notation jf = J$. 
Upon making the pure state ansatz, the resulting simplifications lead to 

T a . a . J = e 0JimrTi $ +f3 ^ ( 6 ) 

To evaluate (||) we now first define the quantities 

N-l 

R^\m) = £ II T °i°^ <W,±1 (7) 

These allow us to derive a 2x2 stochastic recurrence relation, mapping {R^- 1} } onto {R^}: 



rn 



from which the partition sum of (|) follows as 

- lim -LlnR N (m) = - lim -L In [i?f } (m) + (m)] 

N^oo piv N^oo plM 



(8) 



lim — — In < ( J 

iV^oo /3iV 111 



V 



i=2 



^(m) 



(9) 



The successive matrix multiplications above can be simplified via the use of the following ratio 
of the conditioned quantities {R+(m), R^P(m)}: 

2f3mJd} R -\ m ) 



kj " " i?5' } (m) 

It now follows from (g) that these numbers fcj are, in turn, generated by the following stochastic 
process 

k]+1 = ^WW+i+k. e £„^3Vi e 2^^€j (10) 

The stochasticity here is in the pattern components This allows us to work out the 

partition sum and express the asymptotic free energy per neuron as / = extr m /(m), with 

£.£ e{-i,i} p 

(11) 



with 

= Air Z^-W (12) 



j N-l 



The joint distribution (|12|), which is the invariant distribution for the process and which can 
be highly non-trivial Q (depending on the choice of system parameters, the associated integrated 
density can take the shape of a Devil's Staircase), is in practice calculated numerically. In the 
present case one can in fact simplify matters further by exploiting symmetry properties of 
p(k,£,£') resulting from the homogeneous distribution assumed for the 

3.2 Phase Diagrams and Comparison with Numerical Simulations 

We can now extract the macroscopic characteristics of model I by generating the variables 
{kj) Vj < N} numerically (together with the {£f }), which leads us to the joint distribution (12), 
and by subsequently evaluating (numerically) the local minima of the free energy surface (11). 
We show in figure [l] the resulting phase diagrams, for /3J| = {0, —3/2, — 5/2, /3Jf } and for the 
simplest non-trivial case p = 2. In all graphs dashed lines correspond to second-order transitions 
and solid lines to first-order ones. 

Note that for J| = (see figure [j], upper left panel) only pattern \i = 1 is effectively embedded 
in the spin chain, and the phase diagram of the model is identical to that found earlier in || 
for p = 1, as it should. In this diagram we observe, apart from strictly null-recall (N) and recall 
(R) phases, that there is also a region in which the trivial solution and two non-trivial ones 
(one with positive and one with negative m) can be locally stable simultaneously (indicated by 
N 2 )Q. This region corresponds to parameter values for which the two different types of synapses 
compete most strongly (negative nearest-neighbour interactions versus positive infinite-range 
ones). It is separated from the recall region by a second-order transition (dashed line), and from 
the null-recall region by a first-order transition (solid line); these two lines come together at 
{(3Ji, f3Jf} = — | In 3}. Another benchmark solution of || is recovered for the special case 
of having uniform short-range embedding strengths J| = J( (see figure [l], lower right panel). 
Here, two 'pairs' of first-order transition lines have appeared which separate the regions R4 and 
R6 (where m = is unstable and where 4 and 6 m 7^ solutions are possible, depending on 
initial conditions) from the N region. Here the second-order transition line does not touch any 
of the first-order ones. 

The remarkable qualitative difference between the phase diagrams which the aforementioned 
two special cases produce is striking; in our previous study the physical origin of this difference 
was not studied. In particular, although the correctness of the solution had been tested in || 
against extensive numerical simulations, it was not at all clear how and why the second-order 
transition line (dashed line) would change from the exponentially rising curve (figure [T], upper 
left) to the one in the lower right corner of figure H. Our present model generalizes [||, and 
allows for independent tuning of the short-range embedding strengths: one can now bring the 
non-condensed pattern to life in a continuous way. The top right and lower left panels of figure 
|], where Jf 7^ 0, show how the system realises the transition from the p = 1 case to the p = 2 
case (where Jf = J|). The four panels in the figure can be thought of as different cross-sections 



1 From now on regions which allow for locally stable null-recall solutions will be denoted by Ni, with i indicating 
the number of simultaneously locally stable recall solutions. Similarly, regions which do not allow for locally stable 
null-recall solutions will be denoted by Ri. 
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Figure 1: Phase diagram cross-sections of model type I for p — 2, upon making the 'pure state' ansatz 
for pattern \i = 1, with Ji = j[. The parameters Jf and JJ control the short-range embedding strength 
of patterns fi = 1 and /i = 2, respectively, Ji represents the strength of the mean-field interactions, and 
(3 = T _1 is the inverse temperature. In the absence of pattern \x — 2 (upper left) or for equally strong 
short-range embedding strengths (lower right), we recover [H. Solid/dashed lines denote first/second- 
order transitions. Regions R and N represent strictly recall or null-recall regions, whereas Ri and Nj 
correspond to regions where the trivial solution m — is (R) or is not (N) locally stable, and with 
i € {2, 4, 6} giving the number of locally stable m ^ solutions. 



7 



1.0 



0.8 



m. 



0.6 



equi 



0.4 



0.2 



0.0 



* • • • • » • 




.0 -0.5 0.0 0.5 1.0 



_9 o 0|O 



0.0 0.2 0.4 0.6 O.i 



rriimt 



m 



equi 



1.0 



1.0 



0.8 



0.6 



••••••• 



0.4 - 

0.2 - • • *>* % • ' 




-1.0 -0.5 0.0 0.5 1.0 



' 0.0 0.2 0.4 0.6 0.8 1.0 



Figure 2: Simulation results for model type I, for system size N = 1000 and p = 2, showing the 
equilibrium value (i.e. that obtained after 10,000 iterations per spin) of the 'condensed' overlap m cqu i = 
mi(i— >oo) as a function of the initial value Wi n it = mi(t = 0) (solid circles). The initial configurations 
were drawn at random, subject to the constraint imposed by the required value of m\{t = 0). The 
theoretically predicted locations of the ergodic components, as contructed from equation (|i"l|), are also 
shown as local minima of the free energy per neuron in the insets, for comparison. Open diamonds 
represent the equilibrium values of the non-condensed overlaps rri2(t); they are seen to remain zero, 
which justifies a posteriori the 'pure state' ansatz. Left picture: /3J| = —3.5, f3Jf — —4.2, (3Jg = 18. 
Right picture: (3J{ = -5.5, /3J| = -3.5, (3J t = 23. 



of an extended graph in the area {f3J( , /3J| , f3Ji}, which in combination reveal the underlying 
complexity of the model; due to the competing short- and long-range forces and the high degree 
of frustration new regions come to life in parameter space, with multiple locally stable overlap 
solutions. In contrast to the Jf = J| phase diagram, where m = is unstable everywhere, apart 
from the strictly null-recall phase, the other two J| ^ phase diagrams display regions (N2, 
N4 and Nq) where m = coexists as a locally stable state together with multiple locally stable 
m/0 solutions. These latter new phase diagrams appear significantly richer than those found 
in [B], owing to the breaking of the pattern embedding strength symmetry. 

To test and verify our results we have performed extensive simulation experiments. Initial 
configurations cr(t = 0) were chosen randomly, according to 

i 

In figure |2| we plot the equilibrium value mi(t — > 00) of the main order parameter as a function of 
its initial value m\{t = 0) (black circles), in order to probe the existence and location of multiple 
ergodic components. To enable comparison with the theoretically predicted equilibrium values 
we also show (insets) the dependence of the asymptotic free energy per neuron on the order 



parameter m = mi(i — ► 00), as contructed from equations (|ll|) and (16); its local minima are 
indeed located at those values which are found as allowed equilibrium states in the simulations, 
given appropriate initial conditions, and within the experimental margin of accuracy. With our 
system size N = 1000, finite size effects are expected to be of the order of 0{N~z) ~ 0.03. 
Our restriction to relatively small system sizes was prompted by the appearance of extremely 
large equilibration times, due to domain formation. For the case of predominant long-range 
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Figure 3: Simulation results for model type I, for system size N = 1 200 and p = 2, showing the 
'condensed' overlap mi as a function of time (measured in iterations per neuron). The embedding 
strengths were given by /3J| = —3.5, f3J( = —4.2, (3Jg = 18. The relaxation towards zero, following a 
small (but nonzero) initial value, is seen to be extremely slow, due to domain formation. 



interactions equilibration was achieved within ~ 10 4 flips/spin. For predominant short-range 
interactions, however, domain formation led to equilibration times which were observed to scale 
exponentially with the system size, see figure [|. For this reason the observed value of m equ i 
for the ergodic component closest to m = in figure ^ appears to differ from the theoretically 
predicted value m = by roughly 0.06 > 0(N~5). In figure || we show that for the parameter 
choice of figure |2| and for m\(t = 0) = 0.08 the system is indeed approaching the predicted state 
^equii = 0, but extremely slowly. Finally, we have also measured the equilibrium overlaps with 
the non-'condensed' pattern, m,2(t — ► oo), which are seen to remain zero (open diamonds in 
figure |2|), which justifies our 'pure state' ansatz. 



4 Physics of Model II 

4.1 Solution via Random Field techniques 

Neural network models of type II can be solved analytically using the same techniques as applied 
to model type I, although here the calculations will be somewhat more elaborate. Upon again 
making the 'pure state' ansatz: m = (m, 0, . . . , 0) and upon eliminating the conjugate order 
parameters fh via saddle-point equations, we find that the asymptotic free energy per neuron is 
given by / = extr m /(m), with 



f(m) = ^J f m 2 



i}™oojN lnRN{m) 



where the complicated part of the partition sum is in the last term: 

rjV-2 

T,j ia - i+1 cr i+2 

1=1 



R N {m) = 



/3ji 1) (o-jv-i^ JV _ 1 )-(o- J v^ JV ) BJ l m{£ iN ^ 1 cF N ^ 1 +£ iN cF N } 



(13) 
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L (7ia i+ ia i+ 2 



/?ji 1) X; i (^^)-(^+i^ + i)+/3^ 2) E l (^^)-(^+2^+2)+/ 3 ^ m E l ^^ 



(with open boundary conditions). As in model I we next derive a recurrence relation for condi- 
tioned partition sums. For the present model we find that this can be achieved in terms of the 
following four quantities: 

ZV-2 

R^)(rn") - V TT T ffi J « (°N-i£ N -. 1 )i<TN€ N ) p 0Jtm{tN-i<rit-i+SN<rN} X X 

rL ±±\" 1 ) — 11 - t O"iO- i+ iO-j +2 e e °cr JV _ 1 ,±l «CTJV,±1 

<x i=i 

These are found to be successively generated by a 4x4 linear but stochastic iterative process of 
the form Rn+i = T^ + iRn, where Rj = (-R++, R+L,R_+, R-L), the stationary state of which 
will produce the free energy per neuron. The 4x4 random matrix Tjv+i can be decomposed 
further into two coupled 2x2 random matrices: 



R 
R 



(JV+l) 

++ 

(JV+l) 



R 
R 



(JV+l) 













-(3Jim£ N+1 



-PJim£ N+1 



W,4 
-1 
IV, 4 



N- 

-1 

JV,- 



N,— n N,+ 



L 



N. 



L 



N.- 



(14) 
(15) 



where 



L 



N,± 



o 0(Js ^iv— 1 '^N-^^s ^jV-l'^AT+l) 



The partition sum in (|13|) can now be written in terms of the successive multiplication of the 
random matrices T: 



lim — — IuRn 

N^oo (5N 





f 1 ] 




1 


lim — — In < 


N^oo (3N I 


1 




V 1 / 



■ N 
i=3 



R2 



Similar to the analysis performed for model I we again define ratios of conditioned partition 
functions (although here we will need three rather than one): 



k[ 2) 



R 



U) 



k{ 3) 



dO') '" j ' r?(j) ' rO') 

According to (|l4|-|l5|) these ratios are generated by the following stochastic processes: 



A: 



(i) 



J 3 



'J+1 



l(2) 
K i+1 



fr(3) 



J J 



e p4%-A j+1 feWfcW + g-^^-i^+i ofl r(1) 
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The free energy per neuron can now be expressed in terms of the stationary distribution of this 
stochastic process: 



f{ m ) = -J im 2 - lim ~ [dk Y p(fc,£,£') ln(e^ (2) ^' + e-^ 2) ^' k^k^) (16) 

ti e{-i,i}* 



with 



7V-3 



P (fe, e, = lim ^ E - % £ <V f ' ( 17 ) 

4 = 1 



and fc i = (fcf ) ,fcj 2) ,fc? 3) ). 

4.2 Phase Diagrams and Comparison with Numerical Simulations 



Numerical evaluation of the energy surface defined by fllq ) leads to the phase diagrams shown 
in figures |I] and|| which describe the cases p = 1 and p = 5, respectively. They are drawn in the 
{PJ^\pJt} plane, for four different values of the next-nearest neighbour embedding strength 

(2) 

Js . In all phase diagrams the solid lines represent continuous (second-order) phase transitions, 
whereas the dashed lines correspond to discontinuous (first-order) ones. In the absence of next- 

( 21 I I 

nearest second neighbour interactions, i.e. for Jg = (upper left graph in figure f|) our model 
reduces to that of fS. For J s (2) > one finds no new phase regimes, compared to the = 

(2) 

case; the two transition lines of the J s = phase diagram are found to simply move towards 
f3J^ = oo. However, as soon as < 0, frustration effects become more important, with 
new regions appearing in the phase diagram as a result. In the upper right graph of figure 
@, where f3jP = -0.6, we observe that three new regions have been created: region D (with 
m 0), region F (where m = and m ^ are simultaneously locally stable) and region E 
(with two positive and two negative locally stable ra/0 states). In the lower left graph, where 

(2) 

pjr = -o.8, we see that, in addition to the previously created regions, a further new region 
G comes to life, where the trivial state as well as four m/0 ones are all simultaneously locally 
stable. For p > 1, first-order transition lines are found to emerge as boundaries of 'islands' in the 
{PJ^ l \ PJ[\ plane, where four m/0 solutions are simultaneously locally stable. For increasingly 

(2) 

negative values of J s these islands expand in size, and at some point start overlapping, which 
creates additional new regions. In figure || we show typical phase diagrams for the case p = 5. 
Extensive numerical work also shows that increasing the number of patterns p further leads to 
the appearance of further transition lines in the phase diagrams. This is due to the explicit 
dependence of the free energy per neuron, as defined by equation (^), on p. Unlike the more 
conventional long-range Hopfield-type networks (l|, |2|, where after the 'pure state' ansatz has 
been made the macroscopic observables have become independent of p, in the present model the 
short-range interactions ensure that thermal fluctuations around a pure state will always induce 
no-negligible p-dependent interference on the recall of the pure state. 

To test our results we have again performed extensive numerical simulation experiments, the 
results of which are shown in figure ||. The initial states were drawn similar to those in the 
simulations of model type I. We plot the equilibrium overlap oder parameter mi(t — > oo) as a 
function of its initial value m\{t = 0), to probe different ergodic components, and compare the 
locations of these components (in terms of the associated values of m\{t — > oo)) with the 
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Figure 4: Phase diagram cross-sections of model type II for p — 1, with Jg X \ and Jg denoting 
nearest-neighbour, next-nearest neighbour, and long-range embedding strengths, respectively. In the 
absence of next-nearest neighbour interactions (upper left) we recover the model of [pi, whereas for 

(2) 

new regions appear in the phase diagram, with different numbers of simultaneously locally 
stable solutions for the 'overlap' order parameter m. Solid lines denote first-order transitions, dashed 
lines denote second-order ones. 
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Figure 5: Phase diagram cross-sections for model type II, for p = 5 and upon making the 'pure state' 
ansatz, with jj; , and Je denoting nearest- neighbour, next-nearest neighbour, and long-range em- 
bedding strengths, respectively. The two graphs show typical results for the p > 1 phase phenomenology. 
The 'islands' correspond to regions with four simultaneously locally stable states (two with positive to, 
and two with negative to). The structural differences between the p = 1 and p > 1 diagrams can be 
understood upon modulating the embedding strengths of the stored patterns, as with model type I. 
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Figure 6: Simulation results for model II, for system size N = 1 000 and p = 1, showing the equilibrium 
value of the 'condensed' overlap TO equ i = tyl\ it — ► oo) as a function of the initial value TOi n it = toi (t = 0) 
(solid circles). The initial configurations were drawn at random, subject to the constraint imposed 
by the required value of mx(t — 0). The theoretically predicted locations of the ergodic components, 
as constructed from equation (|l^), are also shown as local minima of the free energy per neuron in 
the insets, for comparison. Left picture: = —2.5, (3J^ = —1.2, (3Ji = 12.5. Right picture: 

/3J S (1) = -.5, pji 2) = -1.2, pj e = 6.5. 
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theoretical prediction by also showing (inset graphs) the asymptotic free energy per site as 
constructed from equations ( p"l"D and (^). The agreement between the two is quite satisfactory, 
and well within the error margin given by the finite size effects (with N = 1000 these are 
estimated at 0(N~^) 0.03). 

5 Discussion 

In this paper we presented an equilibrium statistical mechanical analysis of a generalized family 
of recurrent neural network models, with information stored in the form of attractors in the 
neuronal state space, but with ID spatial structure and competition between short-range and 
long-range information processing. We have solved two specific classes of problems. In the first 
class, patterns are embedded in both the long-range and nearest neighbour interactions of the 
neuronal chain, but with pattern-dependent embedding strengths (similar to ||). This general- 
izes a previous study ||], where all embedding strengths were independent of the pattern labels. 
The breaking of the previous embedding strength symmetry is found to yield significantly richer 
phase diagrams, and, moreover, serves to elucidate the remarkable structural differences which 
were observed (but not understood) in || between the phase diagrams for the two simplest cases 
p = 1 and p = 2. In our second class of models, which is a qualitatively different generaliza- 
tion of the models in Q, our neurons are equipped with next-nearest neighbour interactions 
(in addition to the long-range and the nearest-neighbour ones), which increases significantly 
the potential for frustration and competition, given appropriate choices of the various pattern 
embedding strengths. We have been able to solve our models exactly, dealing with the random 
transfer matrix multiplications in the relevant partition sums (generated by the short-range 
interactions) using suitable adaptations of the random-field techniques presented in Q. Al- 
ternatively, one could solve our present models using the random- field techniques of which 
provides an independent theoretical test of our solution: we have carried out this test, and found 
full agreement (see appendix). For both model types we found surprisingly rich phase diagrams, 
with qualitatively distinct topologies, and interesting scenarios of phase diagram deformation 
when appropriate control parameters are varied. Extensive numerical simulation experiments 
support our theoretical results convincingly, in terms of the appearance and location of the 
multiple ergodic components in phase space. 

Note that we have concentrated in this paper on the analysis of the phenomenology of 
dynamical phase transitions, i.e. we have concerned ourselves with the local stability of extremal 
points of the free energy, written as a function of the main order parameter, rather than with the 
actual value of the free energy in the various locally stable states. In the case of large recurrent 
neural networks one is simply not interested in thermodynamic transitions, since the time-scales 
relevant to their operation as associative memories are much smaller than the escape times of 
locally stable states (which diverge with the system size); the locally stable states, and their 
domains of attraction, determine the relevant physical properties. The models and methods 
in this paper can be adapted in a straightforward manner to cover systems with non-binary 
neuron states, or other types of one-dimensional architectures; the inclusion of more distant 
short-range interactions, however, will lead to higher dimensional random transfer matrices, 
and the calculations will become more involved. Alternatively, one could turn to models with 
synchronous rather than sequential dynamics. 

Long-range models, as in Hi, have been of immense value in shaping our understanding of 
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information processing in attractor neural networks, but are far removed from biological reality. 
Our present study emphasizes once more the richness of attractor neural networks with spatial 
structure, and their analytical solvability (at least, within the context of 1 + oo dimensional 
models). Not only can exact solutions be obtained beyond the familiar infinite range models, 
but one can also generalize the relatively simple but solvable 1 + oo dimensional models of 
[]|], increasing again (albeit with small steps) their biological relevance. This allows one to 
investigate further (quantitatively) the significant impact of simple forms of spatial structure on 
information processing via the manipulation of attractors in recurrent neural networks. 
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A Comparison with Rujan's Solution 

An alternative method to calculate the partition sums (|5|) and (|l~3| ) is provided by the random- 
field technique of Q . This requires the result of each of the individual spin summations to be 
written in the form 

model I : £ T a ._„. = JMUW'i+WUW (18) 

for some {hj,Lj}, for aj S {—1, 1}. This transformation allows us to evaluate for TV" — > oo the 
non-trivial part of (||) 

— lim In R(m) = 

N^oo j3N v ' 



lim — — In 



4cosh[/3^(J^fef + i) + PJ(m£ + hi] cosh[/3^(J^fef + i) + 0Jim$ - K 



where in the thermodynamic limit we will assume that the asymptotic distribution of the stochas- 
tic variables {hi} is uniquely generated by the process 

, _1, cosh[/3E M (Jffiff+i) + PJemtl + hj] 

hi+i 2 111 cosh[/3E M (^ercr + i) + 0^ - h liyj 

becomes stationary. 

In a similar fashion one can perform for N —* oo the partition sum in (^) requiring 

model II : £ T aj _ 1(7j(7j+1 = e ^ (1) ^+i+>> (2) ^ (3) ^+i+^-i ( 2 0) 

<Tj_l 

to be true for aj,aj + \ S {—1, 1}. This allows us to write 

_ l im _L m #( m ) = _ i im _Ly m ( 2 4 fi« fig fi« fi« } 

where the quantities fi^|, are obtained iteratively as functions of three stochastically evolving 
variables {hj^, h?_ 1: hj_ 2 ; V j < i}\ 

n8, = coeh[(/^ 1 >&.^ 1 +^ 

where 9{j) = 1 if j > and =0 otherwise, and with 

1 oW nW , , i nW nW , , i n(') nW 

4 "4 ln fi« fi» "4 ln fi« n W 4 "4 ln fi« fi» (21) 

Numerical iteration of the processes (|l9| ) and (^) and subsequently evaluation of the asymptotic 
free energies and of the order parameters shows excellent agreement with the results found earlier 
for models I and II. 
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